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START 



s as s 



T 



Turn display on, using the image 
from the light sensor, find display 
area based on the characteristics 
of the display 6y the known image 
on the display. 



^1 



Turn display off, with the light sensor, capture a frame of the display 



Turn display on. capture a frame of the display space* 



Si 



Locate display by examining the difference between the two frames 



Adjust the light sensor position and sensing parameters for best viewing conditions 




No 



Define the borders of the display in the image captured by the light sensor as continuous lines or curves 



Borders of the display as it is 
captured by the light sensor, their 
visual characteristics, location and 
curvature 



Continue to pointing device 
registration ( Figure 15 ) 
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Instruct user to register the pointing, device he/she will use. The user may select a pointing device from a list or allow the 
system to view the pointing device by allowing the pointing device to be viewed by the light sensor. 
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Instruct user to point the-pointing 
device to various .points in the 
display. Capture one or more 
frames of the display space with 
the pointing device 



Capture a frame of the display space without the pointing device 
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Capture a frame of the display space with the pointing device 
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Locate pointing device by.examining the difference between the two 
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Adjust the tight sensor position and sensing parameters for best viewing conditions 




No 



■ Determine the distinguishing characteristics of the pointing device as opposed to the rest of the display by observing it via 
the light sensor against any image on the display or against a set of calibration images. Adjust the light sensor position and 
sensing parameters for optimum operation. - 
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Continue to computing the mapping 
between the display space registered 
by the light sensor and the computer 
display 
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Distinguishing characteristics 
of the pointing device against 
a variety of display 
backgrounds. 
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START 



Turn display on, instruct the user to 
point the pointing device to a 
convenient location of the display 
(e.g.. center). Using the image 
from the light sensor, find the 
pointing device based on the 
known characteristics of the 
pointing device. 



f 




Instruct user to hide pointing device, with the light sensor, capture a 


frame of the display space 
v ' , 




1 





Instruct user to point the pointing device to a convenient location of the 
display, capture a frame of the display space 



Locate the pointing device by examining the difference between the 
two frames 
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Adjust the light sensor position and sensing parameters for best viewing conditions 




Instruct user to point with the pointing device to the borders and/or various locations of the display. Capture frames with light sensor. 
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Define the borders of the display space in the image .captured by the light sensor as continuous lines or curves 
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Borders of the display as it is 
captured by the light sensor, their 
visual characteristics, location and 
curvature 
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Continue to pointing device 
registration ( Figure 17) 
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Instruct user to point with the pointing device to the borders and/or various locations of the display. Capture frames with light sensor. 



■Adjust the light sensor position and sensing parameters for best viewing conditions 




No 



Determine the distinguishing characteristics of the pointing device as opposed to the rest of the display by observing it via 
the light sensor against the background of the virtual display. Adjust the light sensor position and 

sensing parameters for optimum operation. 



Continue to computing the mapping 
between the display space registered 
by the light sensor and the computer 
display 
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Distinguishing characteristics 
of the pointing device against 
a variety of display 
backgrounds. . 



FlCr 11 



no 



,/40 



13/20 



Borders of the display as it is 
captured by the light sensor, 
their visual characteristics, 
location and curvature 
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Divide the display space into the same number 
of regions as the those of the computer display 
using the information from the borders of the 
display space. 



Distinguishing characteristics 
of the pointing device (against 
a variety of display 
backgrounds if real). 



Establish correspondence between the display 
space observed by the light sensor and the 
computer display region by region. Make 
necessary adjustments to the boundaries of 
individual regions as necessary. 



Images captured with the 
pointing device pointing to 
various regions of the display. 
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Using the information from the position of the 
pointing device previously registered by the 
user and the images captured when the user 
pointed the pointing device to the regions of the 
display as instructed by the system, make 
further adjustments to the mapping between 
the image space registered by the light sensor 
and the computer display. 
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Continue to system operation 



Mapping between the display 
space and the computer 
display 
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Define region around item or icon that 
will be used to determine if a single 
click is warranted 



/ 



Define number of frames or length of 
time that the pointer has to be in the 
region to highlight. the item 
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Find pointer device and position the 
computer mouse accordingly on the 
display, store mouse position is stack 



Examine stored mouse positions to 
determine whether they are all inside the 
bounding box around the item or the icon. 
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Highlight item. 



/ 



Clear stack 




Yes 



Throw out the oldest mouse 
coordinate from the stack. 
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System 
Operation 



Get data from sensor or 
image frame from camera 





r 


Locate pointing device 
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device to point on the real 
or virtual display space 





Find the computer display 
position corresponding to _ 
the pointing device position 




r 




Position the computer's 
pointing icon (e.g., mouse 
arrow) at the computed 
computer display position 
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FIG 28 



